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Abstract 


Learning theory from the 'behaviorist' camp suggests that quick feedback on a stimulus (problem) 
followed by repetition (resubmission) will increase student learning. To test this assumption an 
experiment was conducted. In an introductory management information system class students were 
given the opportunity to submit several skill-building project assignments prior to the due date. These 
submissions were graded promptly and feedback was provided. Students could then re-submit the 
project for final grading upon the actual due date. Data that were collected from a total of 159 
students on three different database and spreadsheet skills indicate that there is a relationship 
between the choice of a student to take advantage of pre-grading and the grade on a subsequent test 
that assesses similar skills as in the project assignments. However, the relationship is not immediate, 
but it appears that students need to follow a 'path of learning' in order to achieve a higher level of 
understanding, whereby prompt and constructive feedback can play an important role. 

Keywords: pedagogy, learning theory, feedback, computer literacy 


1. INTRODUCTION 

As introductory class sizes increase and more 
classes move to the web or a blended delivery 
method, building more learning options 
independent of the instructor are needed. 
Instructors have moved from the 'Sage on the 
Stage' to instructors who need to guide students 
to self-directed learning opportunities. (King, 
1993; Jones, 1999). 


A particular challenge for instructors of 
introductory computer literacy courses is to 
provide the appropriate level of hands-on skill 
assignments with clear feedback, followed by an 
opportunity for the students to learn from their 
errors. Unfortunately, students often only 
receive a grade and some minor comments as 
feedback, and no option is given to correct the 
errors and learn from them. 

Learning theory suggests that increased learning 
will occur with additional stimuli and responses 
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(Gagne, Briggs and Wager, 1992). Even though 
there are several studies in pedagogy and 
psychology disciplines addressing this argument, 
they fail to address the validity of this theory in 
teaching skills. 

This study investigates the results on 'learning' 
of providing students the opportunity to submit 
their assignments (database or spreadsheet) in 
advance of the due date (pre-grading). Students 
then received some high-level of feedback and 
were given the option to re-submit an updated 
assignment prior to the final due date. We 
expected that if students could 'correct' their 
errors before moving on to the next assignment 
or concept, learning from feedback would occur. 
In the current paper, we describe the 
experiment and its results, in an effort to 
address the following research question: Does 
pre-grading followed by prompt feedback 
support student learning? 

In the following, we first provide an overview of 
the relevant literature on learning as a 
background to our study. We then describe the 
experiment, analyze and discuss the data that 
we collected, and draw conclusions. 

2. BACKGROUND 

Student success is influenced by the ability of 
the educator to present new information and to 
evaluate student understanding of the 
information. This process requires the student to 
learn the material covered by the educator. 

Based on behavioral learning theory, Gagne et 
al. (1992) proposed several design principles for 
effective instructional courses, including 
contiguity, repetition, and feedback. Contiguity 
is the concept that the feedback should follow 
the response without delay. The longer the delay 
of the feedback to a learning stimulus, the less 
is the likelihood of correct answers to future 
similar questions. The second principle of 
repetition states that practice strengthens 
learning and improves a learner's retention. 
Gagne et al.'s (1992) conceptual framework of 
cognitive learning includes nine conditions for 
learning: 

1. Gaining attention (reception) 

2. Informing learners of the objective 
(expectancy) 

3. Stimulating recall of prior learning (retrieval) 
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4. Presenting the stimulus (selective 
perception) 

5. Providing learning guidance (semantic 
encoding) 

6. Eliciting performance (responding) 

7. Providing feedback (reinforcement) 

8. Assessing performance (retrieval) 

9. Enhancing retention and transfer 
(generalization) 

The results of subsequent research studies 
suggest that responding (#6) and reinforcement 
(#7) are the events most directly connected to 
student success (Martin, Klein & Sullivan, 2007). 

Murray (1998) encouraged a teaching style 
based on drill/rote learning and memorization, 
whereby modules should be built with many 
exercises that are example-driven. The principle 
of feedback requires that instructors inform the 
learner about whether an answer was correct or 
incorrect. In the case of an incorrect answer, 
feedback should include a new path to solve the 
problem. This new path could be a hint at the 
correct answer, a restatement of a prior fact, or 
even a new example that is less complicated 
(Uden and Beaumont, 2006). In addition, 
feedback to indicate that an answer is correct is 
suggested to be just as important as feedback 
on incorrect answers. 

Orientation and recall is defined as a process 
where learning involves the synthesis of prior 
information that must be recalled to short-term 
memory (Uden and Beaumont, 2006). Similarly, 
there is a school of thought that learners 
construct knowledge by making sense of 
experiences in terms of what is already known 
(Eugenia, 2010). 

In the framework of cognitive learning, 
responding is required from learners after they 
have been given sufficient material to 
comprehend an objective (Tomei, 2008). In 
particular when practice is included in a lesson, 
an active response to the material may be 
expected from the student. For example, 
following a database lesson, responding might 
require a student to create a query that will 
count the number of records in a table in order 
to demonstrate the comprehension of this newly 
introduced concept. 

Given that responding can reinforce students' 
understanding, researchers have suggested that 
effective practice should parallel the 
assessments that are used to test the skills and 
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knowledge reflected in an objective (Reiser and 
Dick, 1996). 

The current study builds on Gagne et al.'s 
(1992) framework. We focus on response and 
reinforcement as key learning components, as 
we investigate how hands-on skills could be 
taught more effectively. We trust that the 
knowledge gained from our study provides 
valuable insight for instructors, particularly those 
teaching online web-based courses. 

3. EXPERIMENTAL DESIGN 

For the experiment, we collected data in five 
sections of an introductory information systems 
course that included a number of computer 
literacy assignments and that were taught by 
two different instructors. Students were given 
the option to submit a number of skill-building 
assignments prior to the due date for pre¬ 
grading. Each assignment represented a new 
concept or advanced computer skill that was 
introduced in class prior to the assignment. 
Specifically, the following three skills provided 
the basis for our dataset: 

1. Create a database with queries and multiple 
relationships between tables, using Microsoft 

Access; 

2. Create a spreadsheet with multiple 
scenarios, using Microsoft Excel Scenario 
Manager; 

3. Create a spreadsheet looking for an optimal 
solution, using Microsoft Excel Solver. 

Following optional pre-grading and re¬ 
submission, a final grade for each project was 
assigned after the project due-date. The learned 
skills were then assessed with a hands-on 
portion of a comprehensive test that was given 
later in the semester. Figure 1 details the steps 
for each skill concept, whereby the shaded areas 
and bold text refer to data points that we 
recorded for the current study. 

For the research model (Figure 2), we use the 
grade on the hands-on test as the main 
dependent variable and representing the level of 
understanding that a student has achieved with 
respect to a certain skill at the end of the course 
module. While we did not administer an entry- 
level test to assess a student's initial level of 
knowledge, we assume that the grade in the 
hands-on test is a good indicator for the extent 
to which a student who completes the course 
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possesses the skills and knowledge that the 
course was intended to provide. 



Figure 1: Teaching and Grading Process 
(Experiment setup) 

In order to address our research question and to 
assess to what extent pre-grading can indeed 
enhance learning, and thus lead to a higher level 
of understanding, we wanted to find out whether 
there is a statistically significant link between 
pre-grading (yes/no) and the result of the 
hands-on test. In addition, we were also 
interested in the role of the final project grade 
as an intermediary step toward the hands-on 
test. Consequently, we analyze our data to test 
the following three hypotheses: 
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HI: Pre-grading (yes vs. no) is associated 

positively with the final project grade. 

H2:The final project grade is associated 
positively with the grade in the hands-on 
test. 

H3: Pre-grading (yes vs. no) is associated 

positively with the grade in the hands-on 
test. 

In order to account for systematic differences 
between sections and instructors, we also 
include the instructor as a control variable in the 
model (Figure 2). 



4. DATA ANALYSIS 


Data were collected from a total of 159 
undergraduate students in an introductory 
information systems course, who were taught 
three computer literacy skills (Access, Scenario 
Manager, and Solver). The students represent a 
total of five sections in fall 2010 and spring 2011 
that were taught by two instructors. One session 
was taught online, all other sessions were taught 
in the classroom. For each student, the collected 
data indicated (1) whether the student had 
taken the opportunity of pre-grading (yes/no) 
for a particular skill assignment; (2) the final 
project grade; and (3) the grade in the 
associated hands-on test. 

To account for individual differences in teaching 
style, course structure, and details on projects, 
tests and grading schemes, we controlled for the 
instructor as a fourth variable in our analysis. 
The differences between the sessions of an 
individual instructor were not included in the 
analysis as these tend to be smaller than the 
differences between individual instructors (an 
assumption that was also confirmed by 
additional data analyses not reported here). 


Each dataset pertained to a particular skill 
concept (Access, Scenario Manager and Solver), 
and was analyzed separately. After an initial 
review of the data we dropped records with 
grades of 0% or 1% for a project and/or hands- 
on test, because we assume that those results 
reflect a conscious choice of the students not to 
submit a particular assignment (e.g., after 
assessing their overall grade-related standing), 
rather than the level of student understanding. 
The resulting sizes of the three data samples 
were thus n = 153, n = 143, and n=144 for the 
Access, Scenario Manager and Solver skills 
respectively. Table 1 provides a descriptive 
summary of the data. Student participation in 
the pre-grading option ranged from 58% to 
85%. 


Table 1: Descriptive Statistics 


Skill Concept 

sample size (n) 

Access 

n = 153 

Scenario 

Manager 

n = 143 

Solver 

n = 144 

Pre-qrad 

inq Yes 

85% 

58% 

74% 

Project 

Min 

42.50 

3.00 

31.00 

Grade 

Max 

100.00 

105.00 

100.00 


Mean 

93.94 

84.28 

92.06 


Std Dev 

10.08 

25.21 

12.02 

Test 

Min 

73.00 

2.00 

8.00 

Grade 

Max 

100.00 

100.00 

100.00 


Mean 

97.22 

84.12 

90.89 


Std Dev 

4.46 

21.92 

12.72 


Subsequent data analysis was performed using 
the structural equation modeling (SEM) 
approach with WarpPLS 2.0 software that 
applies the partial least squares (PLS) technique 
(http://www.scriptwarp.com/warppls). SEM is a 
second generation statistical method that, in 
contrast to regression, allows for the 
simultaneous assessment of multiple 
independent and dependent constructs, 
including multi-step paths (Gefen, Straub, and 
Boudreau, 2000). PLS was considered an 
appropriate method to test the research model 
because there is a broad agreement among 
scholars that PLS is well suited for exploratory 
research and theory development (in contrast to 
theory testing), which is the case in the current 
research study. Given that all of the variables in 
the research model included only one indicator, 
it was not necessary to assess the validity of 
latent variables. Instead, we could immediately 
proceed to test our hypotheses with the 
structural model. 
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As is indicated in Figures 3 to 5, we found 
comparable results for all of the three skill 
concepts of Access (Figure 3), Scenario Manager 
(Figure 4), and Solver (Figure 5). In all three 
datasets, H1 was confirmed at high levels of 
statistical significance, whereas H2 was 
confirmed at high to marginal levels of 
significance. Support for H3 was either marginal 
or non-significant. The instructor variable played 
a significant role in all three datasets. Even 
though the indicators of fit between the model 
and the data were acceptable to very good for 
all three datasets, some R square values, in 
particularly for the test grades were below 10%. 
Details follow: 

Skill 1: Access 

For the first dataset (Access) the model fit with 
the data was very good: 

• Average Path Coefficient (APC)=0.165, P=< 
0.001 

• Average R-Squared (ARS) = 0.126, P=0.002 

• Average Variance Inflation Factor (AVIF) 
= 1.091, Good if < 5 



learning' in the reminder of the paper. H3 was 
not supported for the Access skill. With respect 
to the control variable (instructor), we found the 
association between instructor and project grade 
to be significant below the 1%-level, whereas 
the association between instructor and test 
grade was non-significant (Figure 3). 

Skill 2: Scenario Manager 

For the second dataset (Scenario Manager), the 
model fit with the data was again very good: 

. APC=0.213, P=<0.001 

. ARS = 0.130, P=<0.001 
. AVIF=1.051, Good if < 5 

We found HI and H2 to be supported with paths 
that were statistically significant at or below 1% 
indicating support for the path of learning. H3 
was supported marginally at the 6%-level of 
significance. In addition, we found both paths 
between instructor and project grade and 
between instructor and test grade to be 
significant at below 1% (Figure 4). 



Figure 4: Results for Skill 2: Scenario 
Manager 


Skill 3: Solver 


We found HI to be supported with a significant 
path (p< .01) between grading and project 
grade. H2 was marginally supported with a path 
between project grade and test grade that is 
significant at p=0.08, and we also notice a low R 
square (0.04) for the test grade. It is the 
combination of the two paths (1) pre¬ 
grading/project grade, and (2) project 
grade/test grade that we refer to as the 'path of 


For the third dataset (Solver) the indicators of 
model fit with the dataset were very good for 
two out of three indicators (APC and AVIF), but 
the ARS value was marginal: 

. APC=0.165, P=<0.001 

. ARS = 0.095, P=0.102 

. AVIF=1.045, Good if < 5 
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Again, we observed the path of learning in the 
form of strong support for HI (p=0.01) and 
acceptable support for H2 (p=0.05), whereas H3 
was not supported. And again, the association 
between instructor and project grade was 
significant at below the 1%-level of significance, 
whereas the association between instructor and 
test grade was not significant. As reflected in the 
marginal ARS, the R square values for both 
dependent variables were around 10% (Figure 
5). 



Figure 5: Results for Skill 3: Solver 


5. DISCUSSION 

In all three of our datasets, we found strong 
positive associations between pre-grading (yes 
vs. no) and final project grade (HI) and strong 
to acceptable support for the associations 
between final project grade and test grade (H2). 
The direct associations between pre-grading 
(yes vs. no) and the test grade were weaker if 
they were statistically significant at all (H3). 

As a main outcome of our study, our data 
indicate considerable support for the 'path of 
learning' between pre-grading, project grade, 
and test grade independent of assignment and 
number of students who participated in pre¬ 
grading. It appears that the path of learning 
requires two steps and cannot be shortened: 
Students who submitted an assignment for pre¬ 
grading were as such not more (or less) likely to 
achieve a good grade in the hands-on test than 
students who did not take this opportunity (and 
vice versa). We suggest that students need to 
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take the feedback that is provided in the pre¬ 
grading comments serious, and that they 
subsequently have to make an effort to submit a 
high-quality project for a good final project 
score. It is this intermediate step of learning 
that - according to our data - is associated with 
higher grades in the concluding hands-on tests, 
thus, signaling higher levels of understanding 
and learning. 

Several additional issues, however, warrant 
discussion. First, our results reflect a 
considerable amount of noise in the data, as 
indicated in the limited R square values, in 
particular for the hands-on test results. Although 
we do have statistical support for HI and H2 
that are supported by overall acceptable levels 
of model fit, there appears to be a need to 
examine additional variables for a deeper 
understanding of the domain. 

For example, our data showed some highly 
significant associations between instructor and 
grades, in particular project grades. These 
associations most likely represent differences in 
grading practices but could also be an indicator 
of other factors, such as teaching effectiveness. 
At the beginning of the semester, the two 
instructors who participated in the experiment 
coordinated their skills assignments and hands- 
on tests to some degree to ensure structural 
comparability of the resulting data. Both 
instructors also used the same online system for 
grading (Matthews and Janicki, 2010). Despite 
these interactions, however, a number of 
differences remained, for example regarding the 
structure of the syllabus, individual teaching 
styles, project and test instructions, and grading 
rubrics. The format of delivery also varied as 
one of the five sections was taught online. To 
gain additional insights about the role of the 
instructor, an alternative analysis of the data 
was conducted where we measured the strength 
of moderated links between the instructors on 
the one hand, and the associations between pre¬ 
grading, project grades and test grades (H1-H3) 
on the other hand. The specific purpose of the 
alternative analysis was to find out whether the 
path of learning differed in strength between 
instructors, but the results of the analysis were 
not meaningful. Based on these mixed results, 
we suggest that it is necessary to examine in 
more detail the role of the instructor in future 
studies. 

Another issue to consider is whether we are 
witnessing a situation where the smarter 
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students were the ones who primarily took 
advantage of the path of learning. The data 
show that the seemingly most difficult skill 
based on instructor experience and as measured 
by average grades (Scenario Manager), had the 
lowest percentage of students who submitted 
their projects for pre-grading (Table 1). Beyond 
this result, however, we did not have the 
opportunity to collect data on previous skills and 
the knowledge that students brought into the 
course, or on their overall grade level averages. 
We were surprised to find consistently significant 
associations between pre-grading (a choice 
made by the students) and the grades for 
projects and tests (HI and H2). Given the 
limited statistical evidence that pre-grading 
alone resulted in high test scores (H3), we 
suggest our data to indicate that learning occurs 
along a pre-defined path, largely independent of 
a student's previous knowledge. While we 
cannot answer the question of whether smarter 
students benefitted more (or less) from pre¬ 
grading than students that were less smart, we 
found pre-grading to play an important role 
along the path of learning. We suggest that it 
may be up to the individual instructor to 
encourage all students in a class to take the 
opportunity for pre-grading (followed by efforts 
to submit a high-quality final project), given its 
critical role as part of the learning process. 

6. CONCLUSIONS AND OUTLOOK 

In the current paper, we set out to address the 
research question of whether pre-grading 
followed by prompt feedback can support 
student learning. Based on the data that we 
collected for three different skills that were part 
of an introductory information systems course 
taught by two different instructors, we suggest 
the following answer to our question: yes, pre¬ 
grading can support student learning, as long as 
a student takes the feedback from pre-grading 
seriously and makes an effort to subsequently 
submit a high-quality project. 

Pre-grading alone does not seem to guarantee 
learning, as measured by hands-on test results, 
but pre-grading can help to increase the 
likelihood of a student submitting a high-quality 
project as part of the learning process. We 
suggest that our data provides evidence for a 
path of learning that includes three elements: 
(1) early submission of a project for pre-grading 
and prompt feedback; (2) preparation of a high 
quality project based on the early feedback for 
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final project submission; (3) preparation for 
hands-on test based on feedback on the final 
project. Each step along the path is important to 
help a student learn and achieve a high level of 
understanding (test grade). 

Before we conclude the paper, a couple of 
limitations and avenues for future research 
should be emphasized. As mentioned previously, 
our study is limited in its ability to determine 
exactly how much learning has occurred during 
the course, mainly because of the fact that skills 
were not assessed prior to the course. In 
addition, we could not fully explain the role of 
the instructor and also have no explanation for a 
considerable amount of noise in our data. 

The former limitation means that we have not 
addressed in detail how learning actually occurs 
along the identified path of learning, and what 
factors may be particularly helpful in addition to 
pre-grading. While the focus of the current study 
was on the general role of pre-grading as part of 
the learning process, a better understanding of 
what actually happens along the path of learning 
should be considered an important extension of 
our work. In order to help instructors better 
structure their courses, it would be beneficial to 
have a deep understanding about what types of 
learners pre-grading can best support, as well as 
what groups of students are most prone to 
following the suggested path of learning. In this 
context, and as we discussed above, the role of 
the instructor and possibly other factors, need to 
be explored more deeply as well. 

Another extension of our study could be to 
extend the path of learning to series of related 
assignments and projects. In some cases, more 
than one assignment might be given throughout 
a particular course module. It would be 
interesting to see to what extent the path of 
learning can be traced between assignments, 
which could again be helpful for course 
structuring and course management. 

Lastly, it will be important to explore the 
boundary conditions of our findings and 
determine generalizability and applicability of 
the path of learning to other types of 
assignments, students and learning 
environments. While the results of our analysis 
were comparable across assignments and 
sections, and the path of learning was quite 
evident in the current study, the question 
remains, what factors in particular contributed to 
the similarity of the outcomes, and what factors 
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might have obscured or even obstructed the 
path of learning. Only if we understand not only 
the key aspects of the path of learning but also 
its limitations, can we truly move forward in our 
continued quest to help students learn. 
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